Extraction device, data processing system, and extraction method

ABSTRACT

An extraction device for extracting a sub query to be converted to a program for processing stream data continuously inputted to a database, from a query including instructions, as sub queries, to be issued to a database management system. The extraction device includes: an input unit; an operation unit for calculating the memory increase amount in a case of processing the stream data and the processing time to be reduced for each sub query, and calculating the efficiency by using them; and an extraction unit for selecting at least one sub query whose efficiency is equal to or higher than the lower limit value, integrating the memory increase amount calculated for the selected sub query, and on condition that the integrated memory increase amount is equal to or smaller than the maximum memory increase amount, extracting the selected sub query as a conversion object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119 to JapaneseApplication No. JP 2013-257689 filed Dec. 13, 2013, the entire text ofwhich is specifically incorporated by reference herein.

BACKGROUND

The present invention relates to a data processing system for executinga query and processing stream data and data stored in a database, anextraction device, which is used in the data processing system, forextracting a conversion object to be converted to a program forprocessing the stream data from the query, a method for extracting theconversion object, and an extraction program for causing a computer toexecute the method.

As sensors such as IC tags, IC cards, online services such as onlineshopping and online games, cellular phones and smart phones loaded withGPS (Global Positioning System), and car navigation systems connected toa network become widespread, a large amount of data is generatedcontinuously by these sensors or the like. The large amount of data,which is called stream data, flows on a network and is stored in adatabase for analysis or the like.

The data that is needed is acquired from the database by issuing queriesto a database management system (DBMS) which manages the database andexecuting the queries in the DBMS. The queries can cause the DBMS toexecute processing such as updating, and addition and deletion of thedata in addition to acquisition of the data.

In recent years, due to the effects of global warming and the like, windspeed and rainfall amount suddenly change and just a few minutes ofdelay in evacuation sometimes result in a disaster. In order to identifysuch a sudden change and enable quick evacuation, there is a demand fora high-speed query that is capable of acquiring air flow data andrainfall data in real time.

A large amount of data is stored in the database, and it takes time tosearch for and acquire the data that is needed. It may take severalminutes to several hours, although it depends on data amount andthroughput of a machine. This makes it impossible to cope with a casewhere real time acquisition of the data is required.

BRIEF SUMMARY

In view of the problems, an embodiment of present invention provides anextraction device for extracting, as a conversion object, a sub query tobe converted to a program for processing stream data continuouslyinputted to a database, from a query including one or more instructionsas sub queries to be issued to a database management system for managingthe database, the extraction device comprising: an input unit forreceiving input of the query, a maximum memory increase amount of amemory whose use amount increases with the processing of the streamdata, and a lower limit value of efficiency as processing time per unitmemory increase amount, the processing time being reduced when thestream data is processed by the program compared to a case where thedatabase management system executes the query; an operation unit forcalculating at least one memory increase amount when each sub queryincluded in the query is converted to the program and then the streamdata is processed and the processing time to be reduced when the streamdata is processed compared to the case of execution by the databasemanagement system, and calculating the efficiency by using the at leastone calculated memory increase amount and the processing time to bereduced; and an extraction unit for selecting at least one sub querywhose calculated efficiency is equal to or higher than the lower limitvalue, integrating the memory increase amount calculated for theselected sub query, and extracting the selected sub query as aconversion object on condition that the integrated memory increaseamount is equal to or smaller than the maximum memory increase amount.

An embodiment of the present invention makes it possible to prevent theoccurrence of the memory shortage, and execute the processing by usingone query for the stream data and the data stored in the database.

Embodiments of the present invention provide a device and a methodcapable of preventing occurrence of memory shortage and stoppage of asystem, and performing integrated processing using the same query forstream data processing and execution in a data management system.

In the case of processing stream data, it is required to registerqueries and continuously execute the queries for the stream data that iscontinuously inputted. To this end, the queries, where instructions forthe stream data are described, utilize a stream data processing languagesuch as an SPL (Stream Processing Language) for the description. Such aquery is converted to a program for processing the stream data, called astream data processing code, by compilation, and the processing isexecuted by executing the code.

On the other hand, in the case of processing data stored in a database,since it is not easy to describe a plurality of queries of, for example,searching a plurality of tables held by the database in the same streamdata processing language as above, a database query language such as SQLis utilized.

It has been needed to use the queries described in different languagesfor processing the stream data and the data stored in the database.Thus, in the prior art, efficient processing by using one query for thestream data and the data stored in the database has not been possible.

In the case of processing the stream data, the stream data is oncestored in a memory, and then the stored data is subjected to theprocessing. It is needed to store the stream data in the memory.However, when the stream data is the large amount of data, memoryshortage caused by storing the data may stop a system.

Also, in order to achieve hard real time, it is needed to use anapparatus with strict memory restrictions, such as a client computer andan edge server; thus, the memory shortage tends to occur and systemstoppage also tends to occur.

Accordingly, embodiments of the present invention provide a device and amethod capable of preventing occurrence of memory shortage and enablingthe processing to be efficiently executed to both the stream data andthe data stored in the database by one query.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is an entire configuration diagram of a data processing system ofan embodiment of the present invention;

FIG. 2 is a diagram illustrating hardware configuration of a client anda front end server provided in the data processing system illustrated inFIG. 1;

FIG. 3 is a diagram for describing processing to be performed by thedata processing system illustrated in FIG. 1;

FIG. 4 is a diagram illustrating a state of classifying an SQL graphinto two node groups;

FIG. 5 is a diagram illustrating a state of calculating efficiency foreach path of the SQL graph;

FIG. 6 is a diagram illustrating a state of extracting a node of aconversion object in the SQL graph;

FIG. 7 is a functional block diagram of a client that functions as anextraction device;

FIG. 8 is a diagram illustrating an SPL template;

FIG. 9 is a flowchart illustrating flow of processing performed by theextraction device;

FIG. 10 is a diagram illustrating a query written in SQL;

FIG. 11 is a diagram illustrating sub queries, as nodes, included in thequery written in SQL;

FIG. 12 is a diagram illustrating an example description of a sub queryfor performing pipeline processing and an example description of a subquery for performing non-pipeline processing;

FIG. 13 is a diagram illustrating an example of subjecting a final“Inner join” part to the non-pipeline processing; and

FIG. 14 is a diagram illustrating an example of a table generated by thepipeline processing and the non-pipeline processing and an example ofthe SQL converted to refer to the table.

DETAILED DESCRIPTION

Hereinafter, the present invention will be described along a specificembodiment illustrated in the drawings, but the present invention is notlimited to the embodiment described later. FIG. 1 is an entireconfiguration diagram of a data processing system of an embodiment ofthe present invention. The data processing system includes an IOT(Internet of Things) device 10, a packet communication network 11, aclient computer (hereinafter abbreviated as a client) 12, a front endserver 13, and a back end server 14. Although one each of individualapparatuses is illustrated in FIG. 1, the data processing system mayinclude two or more of the individual apparatuses.

The packet communication network 11 is a communication network fortransmitting and receiving packets in which data is divided, andincludes the Internet. The IOT device 10 is a device, which isconnectable to the Internet, for generating the data and transmitting itto the Internet. Examples of the IOT device 10 include a vehicle, ahealthcare device, and various kinds of sensors installed in factories,schools, or the like. Communication between the IOT device 10 and thepacket communication network 11 is conducted by radio communication suchas Wi-Fi via an access point not illustrated in the figure.

Examples of the healthcare device include, for instance, a scale, a bodycomposition meter, a pedometer, an activity meter, a basal thermometer,and a sphygmomanometer. These devices include a sensor for performingmeasurement to generate measurement data and transmit it to theInternet.

The vehicle includes a GPS for measuring a position, a vehicle speedsensor, various kinds of pressure sensors for a fuel pressure or thelike, and various kinds of temperature sensors for room temperature, theoutside air temperature or the like to transmit measurement data that isdetected and generated by each sensor. The healthcare device measuresweight, the number of steps, body temperature, blood pressure, or thelike and transmits the generated measurement data. Examples of thevarious kinds of sensors installed in factories or the like include, forinstance, sensors loaded on a thermometer, a flowmeter, a manometer, adensitometer, or the like installed in a plant or a device for measuringthem.

The client 12 is a PC or the like used by a user, receives input of aquery to be executed in the front end server 13 used as a firstprocessor and the back end server 14 used as a second processor andvarious kinds of parameters, and performs compilation of the query orthe like. Details of the query, the parameters and the compilation willbe described later. Note that the query is a description of an inquiryto a database provided in the back end server 14 in a databasedescription language such as an SQL.

The front end server 13 receives input of stream data such as a largeamount of measurement data transmitted by the IOT device 10 to thepacket communication network 11, processes the stream data, and outputsthe result to the back end server 14. The front end server 13 receives aprogram called a stream data processing code generated by thecompilation of the query by the client 12, executes the stream dataprocessing code, and subjects the inputted stream data to prescribedprocessing. Thus, the processing of part or the entire query can beexecuted.

The back end server 14 receives the result of the processing by thefront end server 13, and when there is remaining processing to beexecuted in the query, executes the processing, stores the result in thedatabase, and also returns the result to the client 12. Also, the backend server 14 performs the processing of storing the stream data in thedatabase. Therefore, the back end server 14 includes a databasemanagement system (DBMS) for managing the database and executing theprocessing to the database.

The database is constructed in the back end server 14, and thus the backend server 14 may be used as a database server. However, it is notlimited to this and the database may be installed to the outside whereit is accessible from the back end server 14. Also, for the database, adatabase adopting a hierarchical data model indicating data by a treestructure or a relational database for making an inquiry by a logicaloperation or the like may be used. When using the relational database,for the DBMS which manages the data, a relational database managementsystem (RDBMS) is used. The relational database and the RDBMS are notdescribed in detail here because they are well known. Hereinafter, adescription will be given assuming that the RDBMS is to be used.

Here, hardware configuration of each apparatus will be described brieflywith reference to FIG. 2. The IOT device 10, although not shown in thefigure, includes a thermometer having a thermistor or the like whenoutputting temperature data, a speedometer when outputting speed data,and a manometer when outputting pressure data, respectively. For thespeedometer and the manometer, a speed sensor and a pressure sensor areused. The IOT device 10 also includes communication means such as atransmitter for transmitting the measurement data to the Internet.

The client 12 includes a CPU 21, a RAM 22, a graphic controller 23, anda display device 24 that are interconnected via a host controller 20,and a communication interface 26, a hard disk drive (HDD) 27, and aCD/DVD drive 28 that are connected to the host controller 20 via aninput/output controller 25. Also, the client 12 includes a ROM 29connected to the input/output controller 25, and a legacy input/outputdevice having an input/output chip 30.

The host controller 20 connects the RAM 22 and the CPU 21 and thegraphic controller 23 that access the RAM 22 at a high transfer rate.The CPU 21 executes a boot program, an OS, and an extraction program,etc., stored in the ROM 29 or the HDD 27. The CPU 21 may be amultiprocessor capable of parallel processing.

The graphic controller 23 acquires image data generated on a framebuffer provided in the INPUT/OUTPUT CONTROLLER 25 by the CPU 21, anddisplays it on the display device 24. The graphic controller 23 mayinclude the frame buffer therein.

The input/output controller 25 connects the host controller 20 that is arelatively high-speed input/output device, the communication interface26, the HDD 27, and the CD/DVD drive 28. The communication interface 26communicates with the other devices via a network. The HDD 27 stores theOS, the extraction program, an application program, and various kinds ofdata, etc. When the extraction program and the various kinds of data orthe like are recorded on a CD-ROM or a DVD, the CD/DVD drive 28 readsand provides them to the input/output chip 30 via the RAM 22.

To the input/output controller 25, the ROM 29 and a relatively low-speedinput/output device such as the input/output chip 30 are connected. TheROM 29 stores, for example, the boot program for loading the OS from theHDD 27 and activating it, and firmware having initialization informationof a computer and an apparatus or the like recorded thereon. Theinput/output chip 30 connects an input/output device of each unit via aparallel port, a serial port, a keyboard port, a mouse port, or thelike.

The front end server 13 and the back end server 14 have the samehardware configuration, and both may use a blade server. Therefore, onlythe front end server 13 will be described. The front end server 13includes one or more server blades 35 having a CPU 31, a memory 32, anHDD 33 and a communication interface 34, and a housing for housing theone or more server blades 35. The housing includes various kinds ofmodules 36 such as a management module for monitoring operations of theindividual server blades 35 and switching to the other server blade whenabnormality is detected, an I/O module for assigning a LAN adapter orthe like required for communication by the individual server blades 35,and a power module, and a back plane 37 having a connector forinterconnecting the server blades 35 and the various kinds of modules36.

The front end server 13 is capable of increasing the number of serversand improving throughput by connecting an additional server blade to theconnector provided in the back plane 37. When using the back end server14 as a database server, the HDD provided in each server blade of theback end server 14 can be used for the database. Note that the front endserver 13 and the back end server 14 are not limited to the blade serverand other rack mount type server and tower type server or the like maybe used.

With reference to FIG. 3, the processing performed by the dataprocessing system will be described in detail. The stream data iscontinuously transmitted from the TOT device not shown in the figure,and the client 12 receives the query indicated as an input SQL 40 and auser definition input parameter 41 corresponding to it in response toinput from a user. Here, four input SQLs 40 are inputted and four userdefinition input parameters 41 corresponding to them are inputted.

The input SQL 40 includes one or more instructions written in SQL as subqueries. The sub query describes the processing or an operation to thedata. Examples of the operation include, for instance, an operation of,when temperature data associated with each device is inputted as thestream data, extracting only the device for which the temperatureindicates 300° C. or higher. In this case, the temperature may be aone-hour average of 300° C. or higher. Since this is one example, theoperation of the sub query is not limited to this.

As the user definition input parameter 41, for instance, the followingthree parameters can be set. The first parameter is a parameter called adata storage time period, and specifies a time period of the data forstoring the stream data in a memory. Since the stream data iscontinuously inputted as packets, in order to perform the processing, itis needed to section the stream data by the time period or the number ofthe packets and perform the processing. Here, as the parameter, the timeperiod is inputted.

For the time period, for instance, the time such as 1 minute or 10minutes can be specified. When 10 minutes is specified as the timeperiod, after the stream data is inputted and is stored for 10 minutes,the data within the 10 minutes is subjected to the processing. Since adata amount to be stored in the memory at one time increases as thespecified time becomes long, the appropriate time is specified. Insteadof the parameter, a parameter for specifying the number of the packetsmay be inputted.

The second parameter is a parameter called the maximum memory increaseamount. When performing the processing, the data is once stored in thememory and then the stored data is subjected to the processing. Theparameter specifies the maximum value (upper limit value) of the memoryincrease amount to be increased by storing the data.

The third parameter is a parameter called a lower limit value ofefficiency. The input SQL 40 includes one or more sub queries. In thedata processing system, the client 12 compiles the input SQL 40,extracts the sub query for processing the stream data from the one ormore sub queries included in the input SQL 40, and converts only theextracted sub query to generate the stream data processing code.

Upon the extraction, in order to determine which sub query is to beextracted, the processing time when the input SQL 40 is executed in theRDBMS without converting any sub query and the processing time when thestream data processing code is generated and the processing is performedfor each sub query are calculated, they are compared, and the processingtime to be reduced is calculated. When performing the processing by thestream data processing code, since the data is stored in the memory andthe stored data is subjected to the processing, the use amount of thememory increases. Accordingly, an increased amount of the memory due tothe storing operation, that is the memory increase amount, is alsocalculated. The processing time to be reduced is divided by the memoryincrease amount to calculate the processing time to be reduced per unitincrease amount of the memory, and it is defined as the efficiency.Therefore, the efficiency becomes an index that indicates how muchprocessing time is to be reduced and how efficiently the processing isto be performed. The third parameter specifies the lower limit value ofthe efficiency calculated in this way.

In FIG. 3, a compiler 42 for performing the compilation parses the inputSQL 40, and generates a graph expressed by a tree structure in whicheach sub query is each node. In the tree structure, a side to which thestream data is inputted is defined as a root node, and a node at the endis defined as a leaf node. The root node has no parent node connectedthereto, and the leaf node has no child node connected thereto. Thecompiler 42 determines whether or not each node is capable of pipelineprocessing on the basis of a parse result, defines the node capable ofthe pipeline processing as a pipeline processing node (P node), anddefines the node incapable of the pipeline processing as a non-pipelineprocessing node (NP node).

The P node is a node in which a condition for processing the data isfixed and which is capable of immediately processing inputted data andoutputting the processing result to a following child node. The P nodeis capable of processing each packet flowing in the stream data, forinstance, so that it has characteristics that the memory increase amountis small and an effect of reducing the processing time is large.

The NP node is a node which includes an aggregate function ofcalculating an average, a total and the maximum value or the like, andhas a need of storing the data within a fixed time period in the memory.Therefore, the memory increase amount is large and there is no effect ofreducing the processing time; however, since in-memory execution ispossible, the processing time can be reduced compared to the case ofexecuting the input SQL 40 in the RDBMS. In-memory is a technique ofwriting all data to be utilized to a memory and not reading it from adatabase when executing a program.

An example of expressing the input SQL 40 with a tree structure isillustrated in FIG. 4. Among the individual nodes, reference character“P” denotes a P node, and reference character “NP” denotes an NP node.Reference characters “m1” to “m17” described on the lower left denotethe memory increase amount when subjecting the node to stream dataprocessing such as the pipeline processing or the non-pipelineprocessing, and reference characters “e1” to “e17” on the right denotethe processing time to be reduced when performing the stream dataprocessing to the node compared to the case of executing the input SQL40 in the RDBMS. The processing time when executing the input SQL 40 inthe RDBMS is not the processing time when actually executing it but isestimated processing time, and is the processing time derived fromcalculation.

Stream data A is inputted to root nodes 50 and 51, stream data B isinputted to root nodes 52 and 53, stream data C is inputted to a rootnode 54, and the processing is executed. For instance, a result of theprocessing in the root node 50 and a result of the processing in theroot node 51 are outputted to a node 55 which is the child node thereof,and processed by the node 55. Then, a result of the processing in thenode 55 is outputted to the next node sequentially and finally outputtedto a leaf node 57 at the end, and final processing is executed in theleaf node 57. A result of the processing in the leaf node 57 becomes aprocessing result of the input SQL 40, and is stored in the back endserver 14, also sent to the client 12 and displayed or the like.

In order to perform the processing at a high speed, the root nodes whichare P nodes are detected, the individual detected root nodes aresequentially traced, and the nodes are classified into one or more Pnode groups formed only of P nodes before reaching an NP node in thetraced route (path) and one or more NP node groups formed of the othernodes. In FIG. 4, the node group formed only of P nodes in a light colorarea is the P node group, and the node group formed of the other nodesincluding NP nodes in a dark color area is the NP node group. In FIG. 4,two P node groups and one NP node group are illustrated. After theclassification, the memory increase amount and the processing time to bereduced are calculated.

A memory increase amount M is a memory amount required for storing thedata assumed to be needed in order for the node to perform theprocessing to the stream data. Processing time t_(RDBMS) when theprocessing is performed in the RDBMS and processing time t_(stream) whenthe stream data processing, that is the pipeline processing or thenon-pipeline processing, is performed are calculated, and processingtime t_(red) to be reduced can be derived from the difference.Specifically, it can be calculated by the following expression (1).[Expression 1]t _(red) =t _(RDBMS) −t _(stream)  (1)

The memory increase amount M and the processing time t_(red) to bereduced can be calculated on the basis of a data rate of the streamdata, a time period (WINDOW) for the stream data to be subjected to theprocessing, statistical information obtained from the result ofexecuting the query in the past, and information about an operation ofthe query. A specific calculation method thereof will be describedlater. By using the memory increase amount M and the processing timet_(red) to be reduced calculated in this way, the processing time to bereduced per unit memory increase amount, that is efficiency E_(m)(second/byte), is calculated by the following expression (2).

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 2} \right\rbrack & \; \\{E_{m} = \frac{\sum t_{red}}{\sum M}} & (2)\end{matrix}$

This efficiency E_(m) tends to be a large value because the memoryincrease amount is small in the case of including P nodes. That is, theprocessing can be efficiently performed. For the memory increase amountM and the efficiency E_(m), the maximum memory increase amount M_(max)of the parameter inputted by the user and the lower limit value E_(min)of the efficiency are compared. The memory increase amount M is added,and until the integrated value M_(com) reaches the maximum memoryincrease amount M_(max), addition of the node as a conversion objectinto the stream data processing code to perform the pipeline processingor the non-pipeline processing is performed. At the time, it is alsoconfirmed simultaneously that the efficiency E_(m) is not the lowerlimit value E_(min) or lower.

When the efficiency E_(m) becomes the lower limit value E_(min) orlower, it means that the memory increase amount is large for the shortprocessing time t_(red) to be reduced and the efficiency is not improvedso much even when the pipeline processing or the non-pipeline processingis performed. Therefore, when the efficiency becomes the lower limitvalue E_(min) or lower, addition of the node as a conversion object isnot performed. For the nodes not extracted as conversion objects, thepipeline processing or the non-pipeline processing is not performed andthe processing is executed in the RDBMS.

Whether or not the addition of the node as a conversion object isperformed can be evaluated by sequentially selecting the node. As oneexample of the selection method, the node can be selected in a widthpriority order of the P node group. Here, width priority is a method ofpreferentially selecting the node in the same hierarchy starting fromthe root node. First, one root node in the P node group is selected. Theroot node at which position is to be selected first can be predeterminedby setting. Whether or not the root node is a conversion object isevaluated, and the adjacent root node at the same hierarchy is evaluatedsequentially. When there is no more node at the same hierarchy, thechild node at the hierarchy one below is traced. For the child node ofthe lower hierarchy, when the parent node is subjected to the processingin the RDBMS, the child node is also subjected to the processing in theRDBMS. When the evaluation of all the nodes in the P node group isended, evaluation is performed in the same way for another P node group.

When there is not another P node group or the evaluation is ended forall the P node groups, evaluation is performed for NP node groups as inthe case of the P node groups, and conversion objects are extracted.When it is determined to execute the processing in the RDBMS for theroot node, evaluation is not performed for all the nodes in the samepath, and it is determined to execute the processing in the RDBMS forthe nodes. The memory increase amount M for the node added as aconversion object is added to the integrated value M_(com) of the memoryincrease amount at any time.

Referring to FIG. 5, a detail description will be given. First, one ofthe P node groups in the light color area is selected. Here, the P nodegroup constituted of the nodes 50, 51, 52, 55 and 56 is selected. This Pnode group has three paths: a path from the node 50 to the node 55, apath from the node 51 to the node 55, and a path from the node 52 to thenode 56.

First, the node 50 is selected, and the efficiency E_(m) is calculatedas e1/m1 by the expression (2). The e1/m1 is compared with the lowerlimit value E_(min), and when it is equal to or larger than the lowerlimit value E_(min) and also a new integrated value obtained by addingm1 to the integrated value M_(com) of the memory increase amount untilthen is equal to or smaller than the maximum memory increase amountM_(max), the node 50 is added as a conversion object. Then, the node 51and the node 52 are evaluated in the same way. If the node 50 and thenode 51 which are the parent nodes of the node 55 become conversionobjects, the efficiency (e1+e5)/(m1+m5) of the path from the node 50 tothe node 55 and the efficiency (e2+e5)/(m2+m5) of the path from the node51 to the node 55 are calculated, the larger one is defined as theefficiency E_(m), evaluation is performed in the same way, and when acondition is satisfied, the node 55 is added as a conversion object. Insuch a manner, for the node having a plurality of paths, evaluation isperformed using the largest value of the calculated efficiency E_(m).

Since the node 55 has no child node which is a P node, evaluation isperformed in the same way for the node 52 which is the adjacent rootnode. In this way, evaluation is performed in the same way for the node56 and the node 54 in another P node group as well.

Here, while the description is given only for the P node groups,evaluation can be performed in the same way for the NP node groups inthe dark color area. Also, in this case, when it is determined toexecute the processing in the RDBMS for the root node, evaluation is notperformed for all the nodes in the same path, and the processing isexecuted in the RDBMS for all the nodes.

Even when the evaluation is performed in such a manner and all the nodesof all the P node groups illustrated in FIG. 5 and some nodes of the NPnode group are determined to be conversion objects, there is a casewhere the integrated value M_(com) of the memory increase amount has notreached the maximum memory increase amount M_(max). In order to performthe processing at a high speed, it is desirable to convert as many nodesas possible and perform the pipeline processing and the non-pipelineprocessing. Therefore, P nodes are evaluated not only from the root nodeside but also from the opposite leaf node side in the same way, andwhether or not each P node is added as a conversion object isdetermined.

FIG. 6 is a diagram illustrating a state of evaluating P nodes from theroot node side, adding P nodes to the conversion objects, and thenevaluating P nodes from the leaf node side because the maximum memoryincrease amount M_(max) is not reached. All the P nodes in the two Pnode groups are added as conversion objects, and some nodes (light colornodes) in the NP node group are also added as conversion objects. In theevaluation, for the leaf node 57, since it is determined to perform theprocessing in the RDBMS for both high-order nodes connected to the leafnode 57, evaluation is not performed, and it is determined to performthe processing in the RDBMS.

Since the leaf node 57 is a P node and the integrated value M_(com) ofthe memory increase amount has not reached the maximum memory increaseamount M_(max), evaluation is performed in the same way from the leafnode side as well. In FIG. 6, since the integrated value M_(com) has notreached the maximum memory increase amount M_(max) yet even when thememory increase amount calculated for the leaf node 57 is added, theleaf node is added to the conversion objects, and for the high-order NPnodes (dark color nodes) thereof, since E_(m) is equal to or lower thanthe lower limit value E_(min), it is determined not to add these nodesto the conversion objects. For the P node held between the NP nodes,since it is determined to execute the processing in the RDBMS for the NPnodes which are the high-order nodes connected thereto both on the rootnode side and on the leaf node side, evaluation is not performed, and itis determined to execute the processing in the RDBMS as in the case ofthe high-order nodes.

In such a manner, by performing the evaluation also from the leaf nodeside and adding the nodes that can be added to the conversion objects,the processing to the processing result in the RDBMS can be performed inthe memory. By gathering the processing to be performed in the RDBMS atthe center and reducing it as much as possible, the overall processingcan be accelerated.

Referring to FIG. 3 again, the nodes added to the conversion objects areconverted by the compiler 42, and a program (pipeline processingprogram) for executing the pipeline processing and a program(non-pipeline processing program) 43 for executing the non-pipelineprocessing called a stream data processing code are generated. For thecontinuously inputted stream data, by an ETL processing program 44,basic preprocessing such as filtering is performed and processing ofconversion to a form that is easy to use, for example, in the pipelineprocessing program and the non-pipeline processing program 43 isperformed. Also, the ETL processing program 44 performs the processingof storing all the stream data in a second database 46.

The pipeline processing program and the non-pipeline processing program43 are arranged to execute the processing in the order of the node 50and the node 55, for instance, of the P node group illustrated in FIG. 4to FIG. 6, and store the processing result in a first database 45 byperiodical write or write by an execution trigger of a residual SQL 47.

For the input SQL 40, since some sub queries are converted to the streamdata processing code, the sub queries are removed, and the remaining subquery to be executed in the RDBMS, that is the residual SQL 47, isgenerated. The residual SQL 47 is executed in the RDBMS, generates atrigger to start the pipeline processing program and the non-pipelineprocessing program 43, and receives a notice that write to the firstdatabase 45 is ended. Accompanying this, the residual SQL 47 acquiresthe processing result by the pipeline processing program and thenon-pipeline processing program 43 from the first database 45 and thedata stored in the second database 46, and executes query processing.The processing result by the residual SQL 47 is presented to the user oris utilized by an external application.

The client 12 functions as an extraction device for extracting aconversion object and a conversion device for performing conversion tothe pipeline processing program and the non-pipeline processing program43. The function can be achieved by executing an extraction program anda conversion program stored in the HDD 27 by the CPU 21. Here, thesedevices are included in the client 12 which is one apparatus, but can beconfigured as separate apparatuses.

The client 12 includes, as illustrated in FIG. 7, an input unit 60 forreceiving input of the input SQL 40 including one or more sub queries,the maximum memory increase amount M_(max) of the memory whose useamount is increased by the stream data processing, and a lower limitvalue E_(min) of the efficiency, as a functional unit thereof. Theclient 12 also includes an operation unit 61 for calculating the memoryincrease amount when the stream data processing is performed and theprocessing time to be reduced when the stream data processing isperformed compared to the case of performing the processing in the RDBMSfor each sub query, and calculating the efficiency by using the at leastone calculated memory increase amount and the processing time to bereduced.

The client 12 further includes an extraction unit 62 for selecting atleast one sub query whose efficiency calculated by the operation unit 61is equal to or higher than the lower limit value E_(min) of theefficiency that is received by the input unit 60, adding the memoryincrease amount calculated for the selected sub query to the memoryincrease amount calculated and integrated for all the sub queries whenthere are already extracted sub queries, and on condition that thememory increase amount after the addition or the calculated memoryincrease amount when there is not already extracted sub query is equalto or smaller than the maximum memory increase amount M_(max),extracting the selected sub query as a conversion object.

The client 12 includes at least the input unit 60, the operation unit 61and the extraction unit 62, and in addition to them, may further includea graph generation unit 63 for parsing the input SQL 40, and generatingan SQL graph expressed by a tree structure indicating dependency betweennodes, in which each sub query is each node. Also, the client 12 mayinclude a classification unit 64 for determining whether or not eachnode is a P node capable of the pipeline processing of processing theinputted data and outputting the processing result on the basis of theparse result, and with reference to the graph, and classifying the nodesinto one or more P node groups in which the nodes hierarchicallyconnected from the root node are formed only of P nodes and one or moreNP node groups formed of the remaining nodes.

When selecting sub queries, the extraction unit 62 can select them inthe descending order of the efficiency by setting. This is because theprocessing can be performed more efficiently by selecting the subqueries sequentially from the one with the higher efficiency. Theextraction unit 62 is capable of extracting conversion objects byselecting the nodes in the descending order of the efficiency from thenodes whose efficiency is equal to or higher than the lower limit value,giving the nodes classified into the P node group priority, on the basisof a classification result in the classification unit 64.

If the integrated value M_(com) of the memory increase amount has notreached the maximum memory increase amount M_(max), the classificationunit 64 performs classification further into one or more P node groupsin which the nodes hierarchically connected from the leaf node side areformed only of P nodes and the NP node groups formed of the remainingnodes. The extraction unit 62 performs evaluation in the same way forthe nodes classified into the P node groups. That is, the node whoseefficiency E_(m) is equal to or higher than the lower limit valueE_(min) is selected, the memory increase amount calculated for it isadded, whether or not the maximum memory increase amount M_(max) isreached is determined, and when it is not reached, the node is added tothe conversion objects.

By extracting conversion objects in this way, it is made possible toconvert them to the pipeline processing program or the non-pipelineprocessing program in the conversion device thereafter, generate theresidual SQLs where a conversion object portion is eliminated from theinput SQLs, send them to the front end server 13 as the first processorand the back end server 14 as the second processor, and execute theprocessing by using one query to the stream data and the data stored inthe database. Also, by extracting conversion objects and limiting theconversion, it is made possible to prevent occurrence of memory shortagedue to the memory increased by the pipeline processing and thenon-pipeline processing.

Conversion objects can be extracted by predetermining clauses (clausesthat can be prepared by a template) of the SQL and finding out apredetermined node of the SQL. When the node other than the conversionobjects is inputted, an alarm is displayed and DB access can be kept oran error can be displayed. Examples of the node of the SQL include a“select” clause for inquiring the data, a “where” clause, a “group by”clause, an “order by” clause, and a “having” clause. Also, the streamdata processing code can be generated using the template correspondingto the clause of the SQL. FIG. 8 illustrates the template of an SPL tothe SQL of “select A1, agg(A2) as agg_A2 from B where C group by D1, D2having E” as an example of the template.

The processing performed by the client 12 will be described briefly withreference to a flowchart illustrated in FIG. 9. The processing isstarted from step 900, and the input unit 60 receives the input of theinput SQL and the user definition input parameter from the user in step910. They are sent to the graph generation unit 63, and in step 920, thegraph generation unit 63 parses the input SQL and generates the SQLgraph expressed by the tree structure indicating the dependency betweenthe sub queries as illustrated in FIG. 4.

In step 930, the classification unit 64 acquires a processing type ofeach sub query obtained by the parse from the graph generation unit 63,determines whether each sub query is a P node or an NP node from theprocessing type, and classifies each sub query into the one or more Pnode groups and the one or more NP node groups. The classification unit64 delivers the classification result to the extraction unit 62.

In step 940, the operation unit 61 calculates the memory increase amountand the processing time to be reduced when the stream data processing isperformed compared to the case of performing the processing in theRDBMS, for each sub query. Then, the operation unit 61 calculates theefficiency from the calculated memory increase amount and the processingtime to be reduced. This processing by the operation unit 61 may beexecuted before the classification by the classification unit 64 or maybe executed in parallel with the classification.

In step 950, the extraction unit 62 receives the memory increase amountM and the efficiency E_(m) calculated by the operation unit 61, theclassification result of the classification unit 64 and the graph, andselects at least one sub query whose efficiency E_(m) is equal to orhigher than the lower limit value E_(min). When there are the alreadyextracted sub queries, the extraction unit 62 adds the memory increaseamount M calculated for the selected sub query to the memory increaseamount calculated and integrated for all the sub queries. Then, when theintegrated value M_(com) of the memory increase amount after theaddition is equal to or smaller than the maximum memory increase amountM_(max), the extraction unit 62 extracts the selected sub query as aconversion object.

In step 960, whether or not the integrated value M_(com) of the memoryincrease amount has reached the maximum memory increase amount M_(max)is determined. For instance, it is possible to determine that themaximum memory increase amount M_(max) is reached when a residual amountto reach the maximum memory increase amount M_(max) is equal to orsmaller than a fixed amount, and to determine that it is not reachedwhen the residual amount exceeds the fixed amount. When it is determinedthat the maximum memory increase amount M_(max) is reached, theprocessing advances to step 980 to end the processing.

In contrast, when it is determined that the maximum memory increaseamount M_(max) is not reached, advancing to step 970, the extractionunit 62 selects at least one sub query whose efficiency E_(m) is equalto or higher than the lower limit value E_(min) from the side of theleaf node. Then, the extraction unit 62 adds the memory increase amountcalculated for the selected sub query to the integrated value M_(com) ofthe memory increase amount, and when it is equal to or smaller than themaximum memory increase amount M_(max), extracts the selected sub queryas a conversion object. When the evaluation from the leaf node side isended, the processing advances to step 980 to end the processing.

Hereinafter, using a specific example, a classification method, a methodof calculating the memory increase amount and the processing time to bereduced, the pipeline processing program to be generated after theextraction and the residual SQL or the like will be described in detail.FIG. 10 is a diagram illustrating one example of the input SQL 40. TheSQL is the query for making real time coupon issuance processing beexecuted.

Issuing a coupon of a convenience store to a driver who has been drivingfor a long time without taking a rest, is stuck in congestion andfrequently visits the convenience store is taken into consideration. Theinput SQL 40 is the query for extracting a pertinent vehicle. In theinput SQL 40, “car_all_table” is stream data transmitted by the vehicle.The stream data is data transmitted by each vehicle at every 100m/second, and includes a vehicle ID for identifying the vehicle, aposition ID for identifying a current position of the vehicle, andvarious kinds of ECU data outputted from an engine control unit (ECU)that controls an engine of the vehicle or the like. Examples of thevarious kinds of ECU data may include a one-minute average speed, andthe operation time of the engine.

“conv_fav_car_table” is a list for which vehicle information about thevehicles that frequently visit a convenience store is acquired from thestream data accumulated in the database as a past action history.“SLOW_LONG_OP_CARS” is a vehicle ID list of the vehicles traveling at alow speed and having the long engine operation time. Whether or not theoperation time is long can be determined based on whether or not anarbitrary threshold is exceeded. “CONV_STORE_FAV_CAR” is the vehicle IDlist of the vehicles that frequently visit a convenience store.

When the input SQL 40 is parsed, the car_all_table which is the streamdata is inputted, and the SLOW_LONG_OP_CARS for performing theprocessing of extracting the vehicle ID list of the vehicles travelingat a low speed and having the long engine operation time from it isobtained as one node 70. Also, the conv_fav_car_table which is theaccumulated stream data is inputted, and the CONV_STORE_FAV_CAR forperforming the processing of extracting the vehicle ID list of thevehicles that frequently visit a convenience store is obtained asanother node 71. Further, a node 72 for joining the vehicle ID lists isalso obtained. In this way, the SQL graph having the tree structure asillustrated in FIG. 11(a) can be generated.

Whether it is the pipeline processing or the non-pipeline processing isdetermined depending on whether or not it is the node in which acondition in the “where” clause is fixed and which is capable ofperforming the processing when the stream data is inputted andimmediately performing transmission to the following node. In an exampleillustrated in FIG. 12(a), the vehicle ID (car id) of the vehicle at aspeed (speed) of 5 km/h or lower is acquired and it is immediately madeinto a list and transmitted, so that it is the pipeline processing.Here, the “where” clause is a command for specifying the condition whenselecting the data or the like.

In an example illustrated in FIG. 12(b), since AVG (average) that is theaggregate function is included and it is needed to store the data of afixed time period in order to calculate the average, transmission cannotbe immediately performed to the following node. Therefore, theprocessing is the non-pipeline processing. For instance, determinationcan be made depending on whether or not specific characters such as AVGare detected in the “where” clause.

Referring to FIG. 11 again, the SLOW_LONG_OP_CARS is determined as thepipeline processing and extracted as a conversion object, and theconv_fav_car_table is determined as the pipeline processing but is notextracted as a conversion object because the efficiency is low. The node70 is, as illustrated in FIG. 11(b), extracted as a conversion objectand converted to a pipeline processing program 73. In the pipelineprocessing program 73, by the execution, the pertinent vehicle ID isacquired, it is turned to a list and a table is generated. The input SQL40 is rewritten to the SQL for which the processing executed by thepipeline processing program 73 is eliminated. For the rewritten SQL, aresidual SQL 74 for referring to the result processed by the pipelineprocessing program 73 is generated.

In the example illustrated in FIG. 11, two tables are joined on thebasis of the vehicle ID for instance in the node 72, and the tableobtained by joining is sent out; however, when the joining processing isperformed as the non-pipeline processing by the non-pipeline processingprogram, the processing to be executed in the RDBMS is reduced andfurther acceleration is achieved. Accordingly, configuration can be asillustrated in FIG. 13. FIG. 13 illustrates the residual SQL for which afinal “inner join” part indicating the joining processing in the node 72is replaced with the non-pipeline processing. Also, “Inner join” is anoperation of gathering common records present in the two tables into oneand joining the two tables.

When the processing is executed by the pipeline processing program andthe non-pipeline processing program, the pertinent vehicle ID isacquired and the table in which the acquired vehicle ID is listed isgenerated. FIG. 14(a) is a diagram illustrating the table generated insuch a manner. The table includes the vehicle IDs for which theone-minute average speed is 5 km/h and long-time driving is performedfor 120 minutes or longer for WINDOW time. The WINDOW is an inputtedtime period.

FIG. 14(b) is the same as the example of the input SQL illustrated inFIG. 9. When the table as illustrated in FIG. 14(a) is generated by thepipeline processing program or the like, the input SQL is converted tothe residual SQL as illustrated in FIG. 14(c) to refer to the generatedtable. Specifically, from the processing of extracting the vehicle ID inan in-memory manner in FIG. 14(b) to the processing of extracting thevehicle ID from the table (SLOW_LONG_OP_CARS), the description of thepart is rewritten.

The calculation method of the memory increase amount and the processingtime to be reduced will be described in detail. The memory increaseamount is, when the sub query is a node, a data amount that needs to beheld in order to perform the processing of the node. For the P nodes,the memory increase amount is data amount for the time of buffering theinputted stream data. By buffering, a plurality of packets can beprocessed simultaneously. For the NP nodes, the memory increase amountis data amount for the WINDOW time of the inputted stream data. For theprocessing of joining (join) the two tables, the memory increase amountis data amount for which the data inputted from the tables is subjectedto the Cartesian product operation.

The processing time to be reduced is calculated by the expression (1).In the expression (1), the processing time t_(RDBMS) when the processingis performed in the RDBMS for the input SQL can be calculated by thefollowing expression (3). In the expression (3), “t_(scan)” is data scantime for searching for the data, and “t_(op)” is data operation time forperforming data selection and an operation.[Expression 3]t _(RDBMS) =t _(scan) +t _(op)  (3)

The data scan time t_(scan) can be calculated by the followingexpression (4), and the data operation time t_(op) can be calculated bythe following expression 5. In the expression (4), “D_(in)” is a datasize of the inputted data, and “U_(scan)” is the scan time per unit datasize. In the expression (5), “D_(op)” is the data size of the data whichis an operation object, and “U_(op)” is the operation time per unit datasize. The U_(scan) and U_(op) are obtained beforehand by actuallyperforming execution in the RDBMS or performing the stream dataprocessing, and the obtained values are used.[Expression 4]t _(scan) =D _(in) ×U _(scan)  (4)[Expression 5]t _(op) =D _(op) ×U _(op)  (5)

In the expression (4), the inputted data size can be obtained bysequentially executing the following processing (i) to (iii).

(i) When there is processing of joining (join) a plurality of tables,the data size for which the inputted data is subjected to the Cartesianproduct operation.

(ii) When there are a “where” clause and a “having” clause, a filterrate of each condition thereof, that is a probability of being possibleto extract the pertinent data, is obtained beforehand from the data(statistical information) accumulated in the past, and the data sizewhen applying the filter rate to the data after the processing (i) iscalculated. Here, the “having” clause specifies a condition as in thecase of the “where” clause, and is a command used in the aggregatefunction such as AVG.

(iii) The data size when applying a column selection rate selected in“Select” processing to the data after the processing (ii) is calculated.Here, the column selection rate is a value calculated by dividing thenumber of selected columns by the number of the columns of the inputteddata.

In the expression (5), for the data size of the operation object, avalue for which the data size after the processing (i) and the data sizeafter the processing (ii) are added can be used.

A description will be given using a specific example. It is assumedthat, for the input SQL, the input SQL 40 in the example illustrated inFIG. 10 is used. Input data is the stream data (car_all_table), and itis assumed that each vehicle transmits the packets of 100 columns atevery 100 msec, and they are inputted. One column of data is assumed tobe 4 bytes. It is assumed that 1,000 vehicles transmit the packets.Accordingly, a data rate of the stream data is 100 columns×4 bytes×10packets/sec×1000 vehicles=4 MB/sec.

Also, accumulated data (conv_fav_car_table) is, assuming that it is 30columns of data for 10,000 vehicles, 10000 vehicles×30 columns×4bytes=1.2 MB.

The buffering time within the nodes of the stream data, that is thebuffering time (time period) in the P nodes, is assumed to be 1 sec, andthe filter rate obtained from the past statistical information isassumed to be 1%. Note that the filter rate is for the vehicle for whichthe average speed is lower than 5 km/h and the engine operation time islonger than 120 minutes. The unit data scan time in the case ofperforming the processing in the RDBMS is assumed to be 10 sec/MB, andthe unit data operation time in the case of performing the processing inthe RDBMS is assumed to be 2 sec/MB. The unit data scan time in thestream data processing is assumed to be 2 sec/MB, and the unit dataoperation time in the stream data processing is assumed to be 1 sec/MB.The stream data processing is assumed to be memory processing of Java®program.

For the node 70 (SLOW_LONG_OP_CARS) illustrated in FIG. 11(a), eachvalue can be calculated as illustrated in the following (a) to (f).

(a) The memory increase amount is defined as the data amount of theinputted data, and is calculated by the data rate×the buffering time.Namely, it is 4 MB/sec×1 sec=4 MB.

(b) The operation object data size is calculated by the memory increaseamount calculated in the (a)+the memory increase amount×the filter rate.Namely, it is 4 MB+4 MB×0.01=4.04 MB.

(c) The processing time t_(RDBMS) in the case of performing theprocessing in the RDBMS is calculated by the memory increase amountcalculated in the (a)×the unit data scan time in the case of performingthe processing in the RDBMS+the operation object data size calculated inthe (b)×the unit data operation time in the case of performing theprocessing in the RDBMS. Namely, it is t_(RDBMS)=4 MB×10 sec/MB+4.04MB×2 sec/MB=48.08 sec.

(d) The processing time t_(S) in the case of performing the stream dataprocessing is calculated by the memory increase amount calculated in the(a)×the unit data scan time in the stream data processing+the operationobject data size calculated in the (b)×the unit data operation time inthe stream data processing. Namely, it is t_(S)=4 MB×2 sec/MB+4.04 MB×1sec/MB=12.04 sec.

(e) The processing time t_(D) to be reduced is calculated by t_(RDBMS)t_(S). Namely, it is t_(D)=48.08 sec−12.04 sec=36.04 sec.

(f) The data size of the data to be outputted to the node 72 iscalculated by the memory increase amount calculated in the (a)×thefilter rate×the column selection rate. Since the number of the selectedcolumns is 1 and the number of inputted columns is 100, the columnselection rate is 1/100=0.01. Accordingly, the data size is 4MB×0.01×0.01=0.0004 MB (0.4 kB).

For the node 71 (CONV_STORE_FAV_CAR) illustrated in FIG. 11(a), eachvalue can be calculated as illustrated in the following (a′) to (f′).The (C′) to (e′) are calculated by computation expressions the same asthose in the (c) to (e).

(a′) The memory increase amount is the inputted data, and is 1.2 MB.

(b′) The operation object data size is 1.2 MB the same as the (a′).

(c′) The t_(RDBMS) is 1.2 MB×10 sec/MB+1.2 MB×2 sec/MB=14.4 sec.

(d′) The t_(S) is 1.2 MB×2 sec/MB+1.2 MB×1 sec/MB=3.6 sec.

(e′) The t_(D) is 14.4 sec−3.6 sec=10.8 sec.

(f′) The data size of the data to be outputted to the node 72 iscalculated by the memory increase amount of the (a′)×the columnselection rate. Since the number of the selected column is 1 and thenumber of the inputted columns is 30, the column selection rate is 1/30.Accordingly, the data size is 1.2 MB×1/30=0.04 MB (40 kB).

For the node 72 illustrated in FIG. 11(a), each value can be calculatedas illustrated in the following (a″) to (e″). Note that (C″) to (e″) arecalculated by the computation expressions as in the (c) to (e).

(a″) The memory increase amount is the input data for the WINDOW time,and is calculated by the Cartesian product of the data outputted fromthe node 70 for the WINDOW time and the data outputted from the node 71for the WINDOW time. Here, the WINDOW time is assumed to be 10 minutes(600 sec). Accordingly, it is (0.4 kB×600 sec)×(40 kB×600 sec)=5760000kB (5760 MB).

(b″) The operation object data size is 5760 MB the same as the (a″).

(c″) The t_(RDBMS) is 5760 MB×10 sec/MB+5760 MB×2 sec/MB=69120 sec.

(d″) The t_(S) becomes 5760 MB×2 sec/MB+5760 MB×1 sec/MB=17280 sec.

(e″) The t_(D) is 69120 sec−17280 sec=51840 sec.

While the extraction device, the data processing system including theextraction device and the extraction method have been described indetail with reference to the drawings, modification can be made within arange that can be conceived by those skilled in the art, such as otherembodiments, addition, change and deletion, and any aspect is intendedto fall within the scope of the present invention as long as the actionand effect of the present invention are demonstrated. Thus, a clientincluding an extraction device and a conversion device or the like maybe provided as well.

Also, this extraction method may be achieved by an extraction programfor causing a computer to execute processing steps to be executed byindividual functional units, and embodiments of the present inventionare capable of providing the extraction program as well. The extractionprogram may be provided by being stored in a recording medium such as aCD-ROM, a DVD, an SD card and an HDD. Also, it can be stored in acontent server or the like and acquired by being downloaded from thecontent server or the like.

REFERENCE SIGNS LIST

-   -   10 . . . IOT device    -   11 . . . Packet communication network    -   12 . . . Client    -   13 . . . Front end server    -   14 . . . Back end server    -   20 . . . Host controller    -   21 . . . CPU    -   22 . . . RAM    -   23 . . . Graphic controller    -   24 . . . Display device    -   25 . . . Input/output controller    -   26 . . . Communication interface    -   27 . . . HDD    -   28 . . . CD/DVD drive    -   29 . . . ROM    -   30 . . . Input/output chip    -   31 . . . CPU    -   32 . . . Memory    -   33 . . . HDD    -   34 . . . Communication interface    -   35 . . . Server blade    -   36 . . . Module    -   37 . . . Back plane    -   40 . . . Input SQL    -   41 . . . User definition input parameter    -   42 . . . Compiler    -   43, 45 . . . ETL processing program    -   44 . . . Pipeline processing program and non-pipeline processing        program    -   46 . . . First database    -   47 . . . Second database    -   50-57 . . . node    -   60 . . . Input unit    -   61 . . . Operation unit    -   62 . . . Extraction unit    -   63 . . . Graph generation unit    -   64 . . . Classification unit    -   70-72 . . . Node    -   73 . . . Pipeline processing program    -   74 . . . Residual SQL

What is claimed is:
 1. An extraction device for extracting, as aconversion object, a sub query to be converted to a program forprocessing stream data continuously inputted to a database, from a queryincluding one or more instructions as sub queries to be issued to adatabase management system for managing the database, the extractiondevice comprising: an input computing unit to receive an input queryhaving one or more sub queries, a maximum memory increase value toindicate an amount memory by which memory usage may increase whenprocessing the streamed data, and a lower limit value of efficiency toindicate a ratio of reduced processing time to increased memory usage,the reduced processing time indicating a difference between a first timeto process the stream data using the program and a second time toprocess the stream data using the database management system, theincreased memory usage indicating an amount of memory by which memoryusage increases due to processing the stream data using the programcompared to processing the stream data using the database managementsystem; an operation computing unit to calculate, for each sub query inthe input query, at least one memory increase amount corresponding to achange in memory usage when the sub query is converted to the programand the program is used to process the stream data, a processing timereduction corresponding to a difference in a time to process the streamdata using the program and a time to process the stream data using thedatabase management system, and an efficiency by using the at least onecalculated memory increase amount and the processing time reduction; anextraction computing unit to select at least one sub query having acalculated efficiency equal to or higher than the lower limit value,integrate a memory increase amount calculated for the selected subquery, and extract the selected sub query as a conversion object oncondition that the integrated memory increase amount is equal to orsmaller than the maximum memory increase amount; a graph generationcomputing unit to parse the input query and generate a graph expressedby a tree structure having at set of one or more nodes, wherein each subquery is a node in the set of one or more nodes; and a classificationcomputing unit to determine whether each node in the set of one or morenodes is a first type of node which executes pipeline processing byprocessing input data and outputting a processing result or a secondtype of node which does not execute pipeline processing based on aresult of parsing the input query, classify nodes in the set of one ormore nodes into one or more first node groups and one or more secondnode groups, wherein nodes in the first node groups are hierarchicallyconnected from a root node and include only nodes from the set of one ormore nodes which are the first type of node and nodes in the one or moresecond node groups include remaining nodes from the set of one or morenodes, and classify the one or more second node groups into one or morethird node groups and one or more fourth node groups, wherein nodes inthe one or more third node groups are hierarchically connected from aleaf node and only which are the first type of node and nodes in the oneor more fourth node groups include remaining nodes of the one or moresecond node groups, wherein the extraction computing unit selects firstnodes corresponding to sub queries whose efficiencies are equal to orhigher than the lower limit value from nodes classified into the firstnode groups and extracts the first nodes as conversion objects, whereinthe extraction computing unit, subsequent to selecting the first nodesand extracting the conversion objects, if the integrated memory increaseamount has not reached the maximum memory increase amount, selectssecond nodes whose efficiencies are equal to or higher than the lowerlimit value from nodes classified into the third node groups andextracts the second nodes as conversion objects.
 2. The extractiondevice according to claim 1, wherein the extraction computing unitselects the sub query in a descending order of calculated efficiency. 3.The extraction device according to claim 1 wherein the operationcomputing unit calculates the memory increase amount and the processingtime reduction based on a data rate of the stream data, a time period ofthe processing of the stream data, statistical information obtained fromthe result of executing the query in the past, and information about anoperation of the query.
 4. A data processing system to extract, as aconversion object, a sub query to be converted to a program forprocessing stream data continuously inputted to a database, from a queryincluding one or more instructions as sub queries to be issued to adatabase management system for managing the database, the extractiondevice comprising: an input computing unit to receive an input queryhaving one or more sub queries, a maximum memory increase value toindicate an amount memory by which memory usage may increase whenprocessing the streamed data, and a lower limit value of efficiency toindicate a ratio of reduced processing time to increased memory usage,the reduced processing time indicating a difference between a first timeto process the stream data using the program and a second time toprocess the stream data using the database management system, theincreased memory usage indicating an amount of memory by which memoryusage increases due to processing the stream data using the programcompared to processing the stream data using the database managementsystem; an operation computing unit to calculate, for each sub query inthe input query, at least one memory increase amount corresponding to achange in memory usage when the sub query is converted to the programand the program is used to process the stream data, a processing timereduction corresponding to a difference in a time to process the streamdata using the program and a time to process the stream data using thedatabase management system, and an efficiency by using the at least onecalculated memory increase amount and the processing time reduction; anextraction computing unit to select at least one sub query having acalculated efficiency equal to or higher than the lower limit value,integrate a memory increase amount calculated for the selected subquery, and extract the selected sub query as a conversion object oncondition that the integrated memory increase amount is equal to orsmaller than the maximum memory increase amount; a graph generationcomputing unit to parse the input query and generate a graph expressedby a tree structure having at set of one or more nodes, wherein each subquery is a node in the set of one or more nodes; a classificationcomputing unit to determine whether each node in the set of one or morenodes is a first type of node which executes pipeline processing byprocessing input data and outputting a processing result or a secondtype of node which does not execute pipeline processing based on aresult of parsing the input query, classify nodes in the set of one ormore nodes into one or more first node groups and one or more secondnode groups, wherein nodes in the first node groups are hierarchicallyconnected from a root node and include only nodes from the set of one ormore nodes which are the first type of node and nodes in the one or moresecond node groups include remaining nodes from the set of one or morenodes, and classify the one or more second node groups into one or morethird node groups and one or more fourth node groups, wherein nodes inthe one or more third node groups are hierarchically connected from aleaf node and only which are the first type of node and nodes in the oneor more fourth node groups include remaining nodes of the one or moresecond node groups, wherein the extraction computing unit selects firstnodes corresponding to sub queries whose efficiencies are equal to orhigher than the lower limit value from nodes classified into the firstnode groups and extracts the first nodes as conversion objects, whereinthe extraction computing unit, subsequent to selecting the first nodesand extracting the conversion objects, if the integrated memory increaseamount has not reached the maximum memory increase amount, selectssecond nodes whose efficiencies are equal to or higher than the lowerlimit value from nodes classified into the third node groups andextracts the second nodes as conversion objects; a conversion device toconvert a sub query of the conversion objects extracted by the dataprocessing system to a program for processing the stream data andgenerating remaining sub queries not to be converted; a first processorfor executing the converted program, processing the stream data, andoutputting a processing result; and a second processor for executing theremaining queries and processing the processing result and the streamdata stored in the database, the second processor including a databasemanagement system for managing a database.